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Random  matrix  theoretic  approaches  to  sensor  fusion  for  sensing  and  surveillance  in 
highly  cluttered  environments 

Abstract:  Powerful  algorithms  that  are  able  to  detect,  estimate,  and  classify  increasingly  weaker 
signals  buried  in  noise  are  a  critical  technological  component  of  several  important  US  Air  Force 
technologies  such  as  SAR,  MIMO  radar  and  hyperspectral  imaging  among  others.  Advances  in 
VLSI  are  making  sensors  cheaper  and  easier  to  deploy  in  increasing  numbers.  What  limits  our 
ability  to  detect  and  discriminate  weaker  statistical  signatures  from  statistical  clutter  is  not  the 
sensor  count  but  algorithm-independent  statistical  limits  associated  with  the  finite  number  of 
snapshots  over  which  the  effects  of  clutter  can  be  averaged  out. 

In  the  work  supported  by  the  Young  Investigator  Award,  we  have  characterized  the 
fundamental  limits  of  statistical  estimation  and  detection  for  a  variety  of  problems  of  direct 
relevance  to  the  US  Air  Force.  These  are  organized  into  two  thrusts: 

Thrust  1:  Characterization  of  fundamental  limits  and  improved  algorithms  for  detection, 
estimation  and  classification  of  matrix-valued  signals  buried  in  noise  with  missing  data  and. 
Thrust  2:  Characterization  of  fundamental  limits  of  and  improved  algorithms  for  transmission  of 
energy  through  highly  scattering  random  media. 

We  shall  now  highlight  the  successes  achieved  in  the  context  of  these  thrusts. 

Thrust  1:  New  algorithms  and  theory  for  detection  and  estimation  of  matrix¬ 
valued  signals  in  noise 

Impact:  Outcomes  of  this  research  include  precise  quantification  of  the  fundamental  limits  and 
associated  performance  guarantees  for  low-rank-signal  matrix  inference  in  the  presence  of  a 
broad  class  of  noise  distributions  that  goes  well  beyond  the  standard  Gaussian  model  and 
includes  missing  data  settings.  These  insights  have  led  to  the  development  of  new  data-driven 
algorithms  for  low-rank  matrix  denoising  that  provably  outperform  PCA  (or  truncated  SVD) 
based  techniques  and  other  convex  relaxation  based  schemes. 

Motivation:  The  truncated  singular  value  decomposition  (SVD)  of  the  measurement 
matrix  is  the  optimal  solution  to  the  representation  problem  of  how  to  best  approximate  a 
noisy  measurement  matrix  using  a  low-rank  matrix.  What  we  are  really  interested  in  practice  is 
the  (unobservable)  denoising  problem  of  how  to  best  estimate  a  low-rank  signal  matrix  buried 
in  noise.  Improving  the  low-rank  signal  estimate  can  yield  improve  estimates  and  classifier 
performance  for  all  of  the  core  Air  Force  sensing  and  sensing  surveillance  task  that  arise  in 
radar,  STAP,  MIMO  radar  and  hyperspectral  imaging. 

We  have  developed  a  new  algorithm  that  exploits  the  Pi's  results  from  random  matrix  theory 
to  improve  denoising  performance  relative  to  the  truncated  SVD  in  the  moderate  to  low  SNR 
regime.  Our  analysis  reveals  the  fundamental  limits  of  low-rank  signal  matrix  extraction  in  the 
presence  of  noise  and  missing  data. 

Successes:  The  main  contributions  of  this  work  are: 
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1)  Our  analysis  shows  that  the  optimal  algorithm  [1]  should  shrink  the  singular  values  of 
the  data  matrix  in  a  Wiener-filter  like  manner  so  that  larger  singular  values 
(corresponding  to  subspaces  with  higher  SNR)  are  shrunk  less  than  smaller  singular 
values  (corresponding  to  subspaces  with  lower  SNR).  The  algorithm  determines  the 
shrinkage  in  an  entirely  data-driven  manner.  See  Fig.  1  and  Fig  2. 

2)  We  show  that  the  new  algorithm  [2]  dramatically  outperforms  algorithms  by  Candes' 
and  others  [2,3]  which  utilize  singular  value  thresholding  including  the  setting  where 
there  are  missing  or  corrupted  entries  as  can  happen  for  radar  in  cluttered 
environments.  See  Fig  3. 

3)  We  prove  that  matrix  regularization  with  any  convex  penalty  function  (such  as  the 
nuclear  norm)  will  be  suboptimal  to  the  new  algorithm  we  have  developed.  If  the 
penalty  function  is  designed  to  match  the  performance  of  our  algorithm  in  the  low  SNR 
regime,  then  it  will  be  suboptimal  in  the  high  SNR  regime  and  vice  versa;  thus  we  will 
outperform  convex  regularization  techniques  and  that  too  without  introducing 
additional  tuning  parameters.  See  Fig.  3 

4)  A  feature  of  our  algorithm  is  that  it  can  automatically  mitigate  the  effect  of  rank  over¬ 
estimation.  This  will  greatly  help  stabilize  the  performance  of  Air  Force  sensing  and 
surveillance  systems  (such  as  radar  and  STAR)  which  require  adaptive  estimates  of 
subspaces  whose  ranks  evolve  with  time  (due  to  the  complexity  of  the  scene);  see  Fig.  4. 

5)  The  analysis  facilitates  the  precise  characterization  of  Mean  Squared  Error  (MSE) 
tracking  performance  for  direction  of  arrival  estimation  in  the  presence  of  noise  and 
missing  data;  see  Fig.  5. 

6)  We  have  developed  new  algorithms  for  robust  RCA  that  outperform  state-of-the-art- 
algorithms  based  on  the  nuclear  norm;  see  Fig.  6. 

7)  In  collaboration  with  colleagues  at  AFRL,  we  have  developed  new  eigen-analysis  based 
algorithms  for  passive  radar  that  can  detect  significant  weaker  signals  than  using 
classical  correlation  based  techniques.  See  [6,7]. 
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Fig.  1.  The  comparison  of  the  weighting  coefficient  for  the  optimal  new  algorithm  (Red 
Triangles)  with  that  of  the  truncated  SVD  (blue  line)  as  a  function  of  the  subspace  SNR.  We  see 
the  shrinkage  effect  where  higher  SNR  subspaces  are  shrunk  less.  Note  the  thresholding  effect 
where  below  a  critical  SNR  we  discard  the  subspace  since  it  is  "uninformative". 


Fig.  2.  Comparison  of  the  denoising  performance  for  the  optimal  new  algorithm  (Red  Triangles) 
with  that  of  the  truncated  SVD  (blue  line)  as  a  function  of  the  subspace  SNR.  We  see  that  we 
improve  get  dramatic  gains  in  the  low  to  moderate  SNR  regime. 
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Fig.  3.  Comparison  of  the  denoising  performance  for  the  optimal  new  algorithm  (Red  Triangles) 
wit  Candes'  singular  value  thresholding  (SVT)  algorithm  with  two  different  tuning  parameters 
as  a  function  of  the  subspace  SNR.  We  see  that  we  get  dramatic  gains  in  both  low,  medium 
and  high  SNR  regimes. 


Fig.  4.  Here  we  compare  the  denoising  performance  as  function  of  the  estimate  of  the  rank.  The 
plot  shows  that  the  new  algorithm  is  able  to  mitigate  the  effect  of  rank  overestimation  better 
than  the  truncated  SVD  (in  blue  line). 
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(d)  MSE  vs  DOA. 


Fig.  5.  MSE  Performance  of  the  MUSIC  direction-of-arrival  algorithm  in  the  presence  of  noise 
and  missing  data.  In  (a)-(c)  we  compare  the  expression,  derived  in  [4]  with  the  expression  in  the 
literature.  In  (d),  we  derive  a  phase  transition  in  system  performance  that  separates  a  regime 
where  the  MUSIC  algorithm  reliably  works  from  a  regime  where  the  MUSIC  algorithm  fails.  The 
precise  boundary  of  the  phase  transition  depends  in  a  simple  manner  on  the  number  of 
sensors,  number  of  snapshots,  proportion  of  data  observed  as  described  in  [4], 
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(a)  Top;  ground  truth  data.  Y.  Bottom:  observed  data,  (b)  New  algorithm  (20)  with  r  =  1  and  As  = 
A'.  0.0()3.^j.  NRMSE  =  9.5%. 
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(c)  SVT-ba.sed  updates  (17)  with  Al  =  6.5  and  As  =  (d)  SVT-based  updates  (17)  with  Xl  =  300  and  As  = 

0.0035.  NRMSE  =  38.4%.  0.0035.  NRMSE  =  42.3%,. 


Fig.  6.  Performance  of  various  algorithms  for  the  background  subtraction  task.  Upper  right  is 
the  new  algorithm  described  in  [5].  The  bottom  two  panels  show  the  Robust  PCA  algorithm  of 
[2]  for  two  different  choices  of  regularization  parameter. 
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Thrust  2:  New  algorithms  and  theory  for  increasing  transmission  through 

opaque  random  media 

Motivation:  Media  such  as  glass  and  air  are  transparent  because  light  propagates  through  them 
without  being  scattered  or  absorbed.  In  contrast,  materials  such  as  turbid  water,  clouds,  fog 
and  egg  shells  are  opaque  because  the  randomly  arranged  particles  cause  light  to  scatter  in 
random  directions,  thereby  hindering  its  passage.  As  the  thickness  of  a  slab  of  highly  scattering 
random  medium  increases,  this  effect  becomes  more  pronounced,  and  less  and  less  of  a 
normally  incident  light  will  be  transmitted  through,  consequently  limiting  the  use  of  optical 
techniques  for  sensing  and  surveillance  in  such  media. 

This  is  particularly  pertinent  to  the  Air  Force  because  of  the  range  of  operating  conditions 
encountered  by  airborne  assets.  In  the  past  year,  we  have  developed  rigorous  theory  and 
algorithms  that  show  that  orders  of  magnitude  improvements  in  light  transmission  can  be 
achieved  through  such  media.  This  can  lead  to  new  opportunities  and  sensing  capabilities  for 
the  Air  Force. 

We  have  developed  theoretical  and  experimental  methods  to  increase  the  transmission  of  light 
by  orders  of  magnitude  through  turbid  random  media  that  exploit  the  recent  discovery  of 
highly-transmitting  eigen-wavefronts  in  a  random  medium  [1-6].  We  have  developed  general- 
purpose  methods  that  can  be  used  at  optical,  microwave  and  acoustic  frequencies  so  that  they 
can  provide  the  foundation  for  broad  new  capabilities  to  the  Air  Force  while  enabling 
communication,  sensing  and  imaging  technologies  for  inhomogeneous  media  of  all  types. 

Successes:  We  have  made  significant  progress  [7]  in  the  theoretical  and  algorithm  portions  and 
thereby  setting  the  stage  for  actual  experiments.  We  summarize  our  successes: 
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1)  We  have  numerically  analyzed  the  phenomenon  of  the  existence  of  highly  transmitting 
eigen-wavefronts  with  fully  spectral  accurate  simulators  for  2D  scattering  systems  as  in 
Fig.  7  and  have  provided  the  first  numerically  rigorous  confirmation  of  the  bimodal 
shape  of  the  transmission  coefficient  distribution  (see  Fig.  8)  and  the  existence  of  an 
eigen-wavefront  with  transmission  coefficient  close  to  one  for  random  media  with  a 
large  number  of  scatterers. 

2)  We  have  developed  iterative,  physically  realizable  algorithms  for  transmission 
maximization  that  utilizes  backscatter  analysis  to  produce  a  highly  transmitting 
wavefront  in  just  a  few  iterations.  See  Fig.  9  for  a  comparison  of  normally  incident  and  a 
highly  transmitting  eigen-wavefront  and  Fig.  10  for  a  convergence  plot. 

3)  We  have  used  and  extended  random  theory  to  prove  conditions  under  which  perfect 
universal  transmission  is  generically  possible  through  such  random  media.  See  Fig.  11 
and  [9]. 


The  development  of  an  algorithm  for  increasing  transmission  using  backscatter  analysis  is  a 
significant  accomplishment.  Of  particular  importance  is  the  fact  that  the  algorithm  constructs  a 
highly  transmitting  eigen-wavefront  in  a  handful  of  iterations  -  therefore  it  is  possible  to 
increase  transmission  by  orders  of  magnitude  in  covert  settings  where  it  is  not  feasible  to 
measure  the  transmitted  wave  and  where  the  medium  is  dynamic  but  relatively  stationary  for 
tens  of  feedback  cycles. 


•  Periodic  repetition 
with  period  D 


Fig.  7.  Geometry  of  the  scattering  system  considered. 
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X 

Fig. 8.  Empirical  transmission  coefficients  distribution  for  the  system  in  Fig.  2  with  D  =  197  A.,  L  = 
1.2  X  10^  A  with  N  =  14,000  randomly  placed  (non-overlapping)  dielectric  cylinders  of  radius  0.11 
A  and  refractive  index  1.3.  Here  the  mean  inter-scatterer  distance  is  about  6.7  A.  Note  the 
existence  of  many  transmission  coefficients  close  to  one  even  though  the  mean  transmission 
coefficient  is  relatively  small. 
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(b)  Wavefield  produmi  by  the  optimal  wavefront. 

Fig. 9:  Wavefield  plot  of  the  incident-plus-backscatter  wave  corresponding  to  (a)  normally 
incident  and  the  (b)  optimal  wavefront  for  the  setup  in  in  Fig.  1  with  D  =  14  A.,  L  =  5.4  A.  and  with 
50  PEC  cylinders  with  a  radius  of  0.11  A  and  mean  inter-cylinder  distance  of  0.8  A.  The  normally 
incident  wavefront  yields  50%  transmission  while  the  optimal  wavefront  yields  99.95  % 

transmission. 
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Fig. 10:  Transmitted  power  as  a  function  of  the  number  of  iterations  is  shown  for  the  new 
algorithms  developed  in  [7],  Here,  we  placed  430,000  cylinders  with  refractive  index  1.3  in  the 
setup  described  in  Fig.  1  and  achieved  a  26x  improvement  in  transmission  relative  to  a  normally 
incident  wave.  Note  the  rapid  convergence  of  the  algorithm. 
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Fig.  11:  The  transmission  coefficient  distribution  predicted  by  new  theory  versus  numerically  rigorous 
simulations.  See  [9]. 
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